An Introduction to Expectation-Maximization

This notes reviews the basics about the Expectation-Maximization (EM) algorithm, a popular approach to perform model estimation of the generative model with latent variables. We first describe the E-steps and M-steps, and then use finite mixture model as an example to illustrate this procedure in practice. Finally, we discuss its intrinsic relations with an optimization problem, which reveals the nature of E-M. 1 The Expectation and Maximization Consider a generative model with parameter θ. The model generates a set of data D, which comprises two parts: (1) the observed data X, and (2) the latent data Z, which is observed. With this model, the complete likelihood of D is given by p(D|θ) = p(X,Z|θ), (1) and the marginal likelihood of the observed data X is given by p(X|θ) = ∑